GH-49452: [Python] Reintroduce docstring injection for stubfiles by rok · Pull Request #49453 · apache/arrow

rok · 2026-03-04T20:17:43Z

Rationale for this change

Warning: should not be merged before #49259.
See #49452 and #48618

What changes are included in this PR?

Adds a wheel build time script to populate stubfiles with runtime docstrings.

Are these changes tested?

Not yet.

Are there any user-facing changes?

Users will get docstrings.

GitHub Issue: [Python] Reintroduce docstring injection for stubfiles #49452

github-actions · 2026-03-04T20:18:11Z

⚠️ GitHub issue #49452 has been automatically assigned in GitHub to PR creator.

rok · 2026-03-09T19:29:46Z

@raulcd this is ready for review

raulcd · 2026-03-10T10:50:52Z

@github-actions crossbow submit wheel*-cp313-*

github-actions · 2026-03-10T10:53:16Z

Revision: f885111

Submitted crossbow builds: ursacomputing/crossbow @ actions-c1678cd8dd

Task	Status
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp313-cp313t-amd64
wheel-macos-monterey-cp313-cp313t-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-musllinux-1-2-cp313-cp313t-amd64
wheel-musllinux-1-2-cp313-cp313t-arm64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64

raulcd

@rok the new check on the ci/scripts/python_wheel_validate_contents.py doesn't seem to work :(

rok · 2026-03-10T12:50:50Z

@github-actions crossbow submit wheel*-cp313-*

github-actions · 2026-03-10T12:55:05Z

Revision: 10509c4

Submitted crossbow builds: ursacomputing/crossbow @ actions-f3f679ce77

Task	Status
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp313-cp313t-amd64
wheel-macos-monterey-cp313-cp313t-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-musllinux-1-2-cp313-cp313t-amd64
wheel-musllinux-1-2-cp313-cp313t-arm64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64

rok · 2026-03-10T13:30:11Z

@github-actions crossbow submit wheel*-cp313-*

github-actions · 2026-03-10T13:32:42Z

Revision: e4903eb

Submitted crossbow builds: ursacomputing/crossbow @ actions-a33b20ecd0

Task	Status
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp313-cp313t-amd64
wheel-macos-monterey-cp313-cp313t-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-musllinux-1-2-cp313-cp313t-amd64
wheel-musllinux-1-2-cp313-cp313t-arm64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64

rok · 2026-03-10T13:33:09Z

@raulcd the docstring presence check was apparently too strict. I think this is ready for review now.

rok · 2026-03-12T13:06:34Z

@raulcd any we could get this reviewed by end of week?

raulcd

Can we add some documentation about the process?
The CMake option and the stub generation (update_stub_docstrings.py) mainly.

Right now I am not entirely sure I understand the process for update_stub_docstrings.py.

raulcd · 2026-03-12T14:31:01Z

python/scripts/update_stub_docstrings.py

+    Populate pyarrow_pkg with source Python modules and installed binary artifacts
+    so that pyarrow can be imported from the parent directory of pyarrow_pkg.
+    """
+    ext_suffix = sysconfig.get_config_var("EXT_SUFFIX") or ".so"


Is this EXT_SUFFIX env var necessary? doesn't seem to be used anywhere apart from this line

It is used below as:

is_extension = ext_suffix in artifact.name or artifact.suffix == ".pyd"

Sorry, I meant whether it can be something different than the default "*.so". I'm not sure is necessary to be able to override as an env variable, what's the use case?

As per 🤖 :
sysconfig.get_config_var("EXT_SUFFIX") returns the platform-specific extension suffix (e.g. .cpython-313-x86_64-linux-gnu.so on Linux, .cpython-313-darwin.so on macOS, .pyd on Windows). We need it to correctly identify compiled extension modules vs other files in the install directory — a plain ".so" check would miss the full CPython-tagged suffixes.

I'll just try copying everything but .pyi files as this really is unnecessarily complicaated. Sorry.

raulcd · 2026-03-12T14:36:40Z

python/CMakeLists.txt

+if(DEFINED SKBUILD_STATE
+   AND SKBUILD_STATE STREQUAL "wheel"
+   AND NOT CMAKE_SYSTEM_NAME STREQUAL "Emscripten"
+   AND DEFINED ENV{CI}


If I understand this correctly this will default to ON on wheels and only on CI, otherwise is off and we have to turn it on manually, i.e. testing locally?

Is this logic necessary? Should we just remove all that, add the option and enable it as part of the wheel build scripts by setting an env var? We can add PYARROW_REQUIRE_STUB_DOCSTRINGS as a possible option on our pyproject.toml:
PYARROW_REQUIRE_STUB_DOCSTRINGS = {env = "PYARROW_REQUIRE_STUB_DOCSTRINGS", default = "OFF"} similar to PYARROW_BUNDLE_ARROW_CPP

Ok, let me try that. Do we want to enable PYARROW_REQUIRE_STUB_DOCSTRINGS at dev time?

python/CMakeLists.txt

raulcd · 2026-03-12T14:54:57Z

python/scripts/update_stub_docstrings.py

+def _create_importable_pyarrow(pyarrow_pkg, source_dir, install_pyarrow_dir):
    """
-    stubs_dir, build_lib = Path(stubs_dir), Path(build_lib)
+    Populate pyarrow_pkg with source Python modules and installed binary artifacts


What does Populate pyarrow_pkg mean? I am not sure I understand what is pyarrow_pkg here. Is it the source tree? Are we copying the Arrow shared objects around to populate the stubs? Otherwise I am unsure what are we copying/linking. Isn't this what PYARROW_BUNDLE_ARROW_CPP does? Maybe we have to build after CMake runs the copy for PYARROW_BUNDLE_ARROW_CPP?

The idea is that pyarrow_pkg is a folder where we can import pyarrow from, so we can grab runtime docstrings from it. Updating comment to make it clearer.

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>

rok · 2026-03-12T18:04:26Z

Thanks for the review @raulcd, I've pushed some changes, can you check if this now makes sense?

rok · 2026-03-12T20:13:41Z

@kou could you do a pass here too? Especially feedback on CMake changes would be valuable :).

kou

+1

github-actions bot added Component: Python Component: Documentation awaiting committer review Awaiting committer review labels Mar 4, 2026

rok mentioned this pull request Mar 5, 2026

GH-36411: [Python] Use scikit-build-core as build backend for PyArrow and get rid of setup.py #49259

Merged

rok force-pushed the docstring-injection branch from 5713fd4 to 095ee4c Compare March 9, 2026 16:11

github-actions bot removed the Component: Documentation label Mar 9, 2026

rok marked this pull request as ready for review March 9, 2026 17:31

rok requested review from AlenkaF and raulcd as code owners March 9, 2026 17:31

inject docstrings into stubs

2056fec

rok force-pushed the docstring-injection branch from 095ee4c to 2056fec Compare March 9, 2026 17:37

rok requested review from assignUser, jonkeane and kou as code owners March 10, 2026 00:34

check stubs are included at wheel build time

f885111

rok force-pushed the docstring-injection branch from 2f0f841 to f885111 Compare March 10, 2026 00:57

raulcd reviewed Mar 10, 2026

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting committer review Awaiting committer review awaiting changes Awaiting changes labels Mar 10, 2026

rok force-pushed the docstring-injection branch from d638b95 to 10509c4 Compare March 10, 2026 12:49

we don't have docstrings yet

e4903eb

rok force-pushed the docstring-injection branch from 10509c4 to e4903eb Compare March 10, 2026 13:18

rok linked an issue Mar 10, 2026 that may be closed by this pull request

[Python] Reintroduce docstring injection for stubfiles #49452

Open

rok mentioned this pull request Mar 11, 2026

GH-49273: [Python] Move stub docstring script to _build_utils with gr… #49357

Open

raulcd reviewed Mar 12, 2026

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 12, 2026

Apply suggestion from @raulcd

f3869c7

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 12, 2026

review feedback

b34e6e1

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 12, 2026

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 12, 2026

review feedback

e7e51db

rok force-pushed the docstring-injection branch from 5576c51 to e7e51db Compare March 12, 2026 21:45

reduce the PR somewhat

a1b43d8

rok force-pushed the docstring-injection branch from fab8f56 to a1b43d8 Compare March 12, 2026 22:36

kou approved these changes Mar 13, 2026

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Mar 13, 2026

Conversation

rok commented Mar 4, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Mar 4, 2026

Uh oh!

rok commented Mar 9, 2026

Uh oh!

raulcd commented Mar 10, 2026

Uh oh!

github-actions bot commented Mar 10, 2026

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

rok commented Mar 10, 2026

Uh oh!

github-actions bot commented Mar 10, 2026

Uh oh!

rok commented Mar 10, 2026

Uh oh!

github-actions bot commented Mar 10, 2026

Uh oh!

rok commented Mar 10, 2026

Uh oh!

rok commented Mar 12, 2026

Uh oh!

raulcd left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Mar 12, 2026

Uh oh!

rok commented Mar 12, 2026

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rok commented Mar 4, 2026 •

edited by github-actions bot

Loading

raulcd left a comment •

edited

Loading

rok Mar 12, 2026 •

edited

Loading